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Highlights 

Highlights  of  GAO-1 1  -502,  a  report  to 
congressional  committees 


DOD  WEAPON  SYSTEMS 

Missed  Trade-off  Opportunities  During  Requirements 
Reviews 


Why  GAO  Did  This  Study 

The  Weapon  Systems  Acquisition 
Reform  Act  of  2009  (WSARA)  directed 
the  Joint  Requirements  Oversight 
Council  (JROC)  to  ensure  trade-offs 
among  cost,  schedule,  and 
performance  objectives  are 
considered  as  part  of  its  requirements 
review  process.  WSARA  also  directed 
GAO  to  assess  the  implementation  of 
these  requirements.  This  report 
addresses  (1)  the  extent  to  which  the 
JROC  has  considered  trade-offs  within 
programs,  (2)  the  quality  of  resource 
estimates  presented  to  the  JROC,  and 
(3)  the  extent  to  which  the  JROC  is 
prioritizing  requirements  and 
capability  gaps.  To  do  so,  GAO 
analyzed  requirement  documents 
reviewed  by  the  JROC  in  fiscal  year 
2010,  which  identified  capability  gaps 
or  performance  requirements  for  new 
major  defense  acquisition  programs. 
GAO  also  assessed  resource  estimates 
presented  to  the  JROC  against  best 
practices  criteria  in  the  GAO  Cost 
Estimating  and  Assessment  Guide. 

What  GAO  Recommends 

GAO  recommends  that  the  JROC 
establish  a  mechanism  to  review 
analysis  of  alternatives  (AOA)  results 
earlier  in  the  acquisition  process, 
require  higher  quality  resource 
estimates  from  requirements 
sponsors,  prioritize  requirements 
across  proposed  programs,  and 
address  potential  redundancies  during 
requirements  reviews.  The  Joint  Staff 
partially  concurred  with  GAO’s 
recommendations  and  generally 
agreed  with  their  intent,  but  differed 
with  GAO  on  how  to  implement  them. 


View  GAO-1 1-502  or  key  components. 
For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


What  GAO  Found 

The  JROC  considered  trade-offs  made  by  the  military  services  before 
validating  requirements  for  four  of  the  seven  proposed  programs  it  reviewed 
in  fiscal  year  2010.  According  to  DOD  officials,  the  most  significant  trade-offs 
are  made  by  the  military  services  during  the  AOA,  which  occurs  between  the 
JROC’s  review  of  an  Initial  Capabilities  Document  (ICD)  and  its  review  of  a 
Capability  Development  Document  (CDD).  The  AOA  is  intended  to  compare 
the  operational  effectiveness,  cost,  and  risks  of  a  number  of  alternative 
potential  solutions.  The  JROC  does  not  formally  review  the  trade-off  decisions 
made  as  a  result  of  an  AOA  until  it  reviews  a  proposed  program’s  CDD.  As  a 
result,  the  JROC  does  not  have  an  opportunity  to  provide  military  advice  on 
trade-offs  and  the  proposed  solution  before  it  is  selected,  and  a  significant 
amount  of  time  and  resources  can  be  expended  in  technology  development 
before  the  JROC  gets  to  formally  weigh  in. 


AOA’s  Relationship  to  JROC  Requirements  Reviews 


JROC  review 


Too  early 
for  trade-offs 


JROC  review 

Requirements 
rarely  change 


Requirements  development  |  |  Acquisition  activity  Acquisition  milestone 

Source:  GAO  analysis  of  DOD  policy. 

The  military  services  did  not  consistently  provide  high-quality  resource 
estimates  to  the  JROC  for  proposed  programs  in  fiscal  year  2010.  GAO  found 
the  estimates  presented  to  the  JROC  were  often  unreliable  when  assessed 
against  best  practices  criteria.  In  most  cases,  the  military  services  had  not 
effectively  conducted  uncertainty  and  sensitivity  analyses  or  examined  the 
effects  of  changing  assumptions  and  ground  rules,  all  of  which  could  further 
the  JROC’s  efforts  to  ensure  that  programs  are  fully  funded  and  provide  a 
sound  basis  for  making  cost,  schedule,  and  performance  trade-offs. 

The  JROC  does  not  currently  prioritize  requirements,  consider  redundancies 
across  proposed  programs,  or  prioritize  and  analyze  capability  gaps  in  a 
consistent  manner.  As  a  result,  the  Joint  Staff  is  missing  an  opportunity  to 
improve  the  management  of  DOD’s  joint  portfolio  of  weapon  programs. 
According  to  Army,  Air  Force,  and  Navy  officials,  having  a  better 
understanding  of  warfighter  priorities  from  the  JROC  would  be  useful  to 
inform  both  portfolio  management  efforts  and  service  budgets.  A  DOD  review 
team  examining  the  JROC’s  requirements  review  process  is  considering 
changes  that  would  address  the  prioritization  of  requirements  on  a 
departmentwide  basis. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


June  16,  2011 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Howard  P.  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

With  the  prospect  of  slowly  growing  or  flat  defense  budgets  for  years  to 
come,  the  Department  of  Defense  (DOD)  must  get  better  returns  on  its 
weapon  system  investments  and  find  ways  to  deliver  more  capability  to 
the  warfighter  for  less  than  it  has  in  the  past.  In  this  environment,  DOD’s 
capacity  to  make  effective  trade-offs  among  cost,  schedule,  and 
performance  objectives  when  developing  and  validating  weapon  system 
requirements  and  acquiring  those  systems  will  be  important  to  achieving  a 
balance  between  DOD’s  weapon  system  investments  and  resources 
available  to  it. 

The  Weapon  Systems  Acquisition  Reform  Act  of  2009  (WSARA)  took 
several  steps  to  encourage  DOD  to  engage  in  a  more  robust  discussion  of 
trade-offs  among  cost,  schedule,  and  performance  objectives  before 
beginning  a  new  weapon  system  program.  First,  WSARA  directed  DOD’s 
Joint  Requirements  Oversight  Council  (JROC),  which  validates  joint 
military  requirements,  to  ensure  trade-offs  among  cost,  schedule,  and 
performance  objectives  are  considered  as  part  of  its  process  for  assessing 
and  prioritizing  requirements.1  Additionally,  WSARA  stated  that  the 
requirements  development  process  must  be  structured  to  enable 
incremental,  evolutionary,  or  spiral  acquisition  approaches,2  and  that 
acquisition,  budget,  and  cost  estimating  officials  should  be  provided  an 


xPub.  L.  No.  111-23,  §  201(b),  (codified  at  10  U.S.C.  §  181  (b)(1)(C)). 
2Pub.  L.  No.  111-23,  §  201(a)(2). 
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opportunity  to  develop  resource  estimates  and  raise  cost  and  schedule 
issues  before  performance  objectives  are  established.3  Finally,  WSARA 
amended  the  U.S.  Code  to  require  the  JROC  to  seek  and  consider,  among 
other  things,  the  input  of  combatant  commanders  when  assisting  the 
Chairman  of  the  Joint  Chiefs  of  Staff  in  identifying,  assessing,  and 
approving  joint  military  requirements;  considering  trade-offs;  and 
establishing  and  assigning  priority  levels,  among  other  areas.4 

WSARA  also  directed  GAO  to  assess  the  implementation  of  these 
requirements.5  This  report  addresses  (1)  the  extent  to  which  the  JROC  has 
considered  trade-offs  among  cost,  schedule,  and  performance  objectives 
within  programs;  (2)  the  quality  and  effectiveness  of  efforts  to  estimate  the 
level  of  resources  needed  to  fulfill  joint  military  requirements;  and  (3)  the 
extent  to  which  the  JROC  is  prioritizing  requirements  and  capability  gaps. 
In  addition,  the  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal 
Year  2011  requires  GAO  to  conduct  a  comprehensive  review  of  the  JROC’s 
requirements  validation  process  and  report  its  results  in  2012.6 

To  conduct  our  work,  we  focused  on  JROC  activities  in  fiscal  year  2010. 

We  chose  this  time  period  to  allow  for  any  changes  the  JROC  would 
implement  as  a  result  of  the  enactment  of  WSARA  in  May  2009.  To 
determine  the  extent  to  which  the  JROC  has  considered  cost,  schedule, 
and  performance  trade-offs  within  programs,  we  reviewed  the  seven 
Capability  Development  Documents  (CDD)  submitted  to  the  JROC  in 
fiscal  year  2010,  JROC  decision  memos  related  to  the  CDDs,  and  analyses 
of  alternatives  (AO A)  conducted  by  the  military  services  prior  to  JROC 
reviews.  We  focused  on  CDDs  because  they  are  the  first  requirements 
documents  that  contain  cost,  schedule,  and  performance  objectives. 
Additionally,  we  reviewed  documentation  from  15  JROC  reviews  of 
programs  that  incurred  substantial  cost  growth  after  program  start  to 
determine  if  cost,  schedule,  and  performance  trade-offs  were  made.  To 
determine  the  quality  and  effectiveness  of  efforts  to  estimate  the  level  of 


3Pub.  L.  No.  111-23,  §  201(a). 

4Pub.  L.  No.  111-23,  §  105(b),  (codified  at  10  U.S.C.  §  181(d)(2)). 

5Pub.  L.  No.  111-23,  §  105(c).  GAO,  Defense  Management:  Perspectives  on  the  Involvement 
of  the  Combatant  Commands  in  the  Development  of  Joint.  Requirements,  GAO-1 1-527R 
(Washington,  D.C.:  May  2011)  reports  on  the  extent  to  which  the  JROC  solicits  and 
considers  input  from  combatant  commanders. 

6Pub  L.  No.  111-383,  §  862. 
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resources  needed  to  fulfill  joint  military  requirements,  we  assessed  the 
resource  estimates  used  to  support  the  7  CDDs  presented  to  the  JROC  for 
approval.  To  do  so,  we  applied  GAO’s  best  practice  criteria  for  cost 
estimates  and  reviewed  supporting  documentation.7  Each  program  was 
also  provided  with  a  copy  of  our  assessment  of  their  resource  estimates 
for  review  and  comment.  To  determine  the  extent  to  which  the  JROC 
prioritized  requirements  and  capability  gaps,  we  reviewed  the  13  Initial 
Capabilities  Documents  (ICD)  and  7  CDDs  submitted  to  the  JROC  in  fiscal 
year  2010,  and  any  discussions  of  priorities  and  redundancies  contained  in 
each  document.  We  also  interviewed  officials  from  DOD,  the  Joint  Staff, 
and  military  service  headquarters  about  the  extent  to  which  the  JROC  and 
its  supporting  bodies  have  addressed  prioritization  issues. 

We  conducted  this  performance  audit  from  June  2010  to  June  2011  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  based  on  our  audit 
objectives.  Appendix  I  contains  detailed  information  on  our  scope  and 
methodology. 


Background 


DOD  uses  three  interrelated  processes  to  deliver  capabilities  to  the  U.S. 
military:  the  Joint  Capabilities  Integration  and  Development  System 
(JCIDS),  which  validates  gaps  in  joint  warfighting  capabilities  and 
requirements  that  resolve  those  gaps;  the  Defense  Acquisition  System, 
which  develops  and  fields  weapon  systems  to  meet  these  requirements; 
and  the  Planning,  Programming,  Budgeting  and  Execution  process,  which 
allocates  the  funding  needed  to  develop,  acquire,  and  field  these  weapon 
systems.  The  JCIDS  process  is  overseen  by  the  JROC,  which  supports  the 
Chairman  of  the  Joint  Chiefs  of  Staff  in  advising  the  Secretary  of  Defense 
on  joint  military  capability  needs.  The  JROC  is  chaired  by  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff,  and  includes  one  senior  leader  from 
each  of  the  military  services,  such  as  the  Vice  Chief  of  Staff  of  the  Army  or 
the  Vice  Chief  of  Naval  Operations. 


7GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developi  ng  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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The  JROC  has  a  number  of  statutory  responsibilities  related  to  the 
identification,  validation,  and  prioritization  of  joint  military  requirements. 
The  JROC  assists  the  Chairman  of  the  Joint  Chiefs  of  Staff  with  a  number 
of  tasks,  including  (1)  identifying,  assessing,  and  approving  joint  military 
requirements;  (2)  establishing  and  assigning  priority  levels  for  joint 
military  requirements;  and  (3)  reviewing  the  estimated  level  of  resources 
required  to  fulfill  each  requirement  and  ensuring  that  the  resource  level  is 
consistent  with  the  requirement’s  priority.  The  JROC  also  assists 
acquisition  officials  in  identifying  alternatives  to  any  acquisition  programs 
that  experience  significant  cost  growth. 

Since  2008,  Congress  has  added  to  the  JROC’s  statutory  responsibilities 
and  increased  the  number  of  JROC  members  and  advisors  who  provide 
input  to  it.  The  National  Defense  Authorization  Act  for  Fiscal  Year  2008 
amended  the  U.S.  Code  to  require  that  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (USD  AT&L),  the  Under  Secretary 
of  Defense  (Comptroller),  and  the  Director  of  the  Office  of  Program 
Analysis  and  Evaluation  serve  as  advisors  to  the  JROC  on  matters  within 
their  authority  and  expertise.8  In  2009,  WSARA  expanded  the  role  of  the 
JROC  by  directing  it  to  assist  the  Chairman  of  the  Joint  Chiefs  of  Staff  in 
(1)  ensuring  that  trade-offs  among  cost,  schedule,  and  performance 
objectives  are  considered  for  joint  military  requirements;  and  (2) 
establishing  an  objective  period  of  time  within  which  an  initial  operational 
capability  should  be  delivered.  WSARA  also  stated  that  the  newly 
constituted  Director  of  Cost  Assessment  and  Program  Evaluation  (CAPE) 
would  advise  the  JROC.9  The  Ike  Skelton  National  Defense  Authorization 
Act  for  Fiscal  Year  2011  allowed  the  Vice  Chairman  of  the  Joint  Chiefs  of 
Staff  to  direct  senior  leaders  from  combatant  commands  to  serve  as 
members  of  the  JROC  when  matters  related  to  the  area  of  responsibility  or 
functions  of  that  command  are  under  consideration.  It  also  added  the 
Under  Secretary  of  Defense  for  Policy,  the  Director  of  Operational  Test 
and  Evaluation,  and  other  civilian  officials  designated  by  the  Secretary  of 
Defense  as  advisors  to  the  JROC  on  issues  within  their  authority  and 
expertise.10 


8Pub.  L.  No.  110-181,  §  942(d). 

9Pub.  L.  No.  111-23,  §  201  (b)  and  (c). 

10Pub.  L.  No.  111-383  §  841. 
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The  JROC  is  supported  in  the  JCIDS  process  by  two  Joint  Capabilities 
Boards  (JCB)  and  seven  Functional  Capabilities  Boards  (FCB),  each  of 
which  is  chaired  by  a  general/flag  officer  or  civilian  equivalent.  JCBs  and 
FCBs  are  responsible  for  specific  Joint  Capability  Areas,  such  as  Force 
Protection,  Logistics,  or  Battlespace  Awareness.  The  JCBs,  FCBs,  and 
associated  FCB  Working  Groups  review  requirements  documents  prior  to 
JROC  reviews.  The  JCB  also  serves  as  the  validation  authority  for 
requirements  documents  that  are  not  associated  with  major  defense 
acquisition  programs  (MDAP).11  In  some  instances,  the  JROC  will  not  meet 
in  person  to  approve  requirements  documents  if  there  are  no  outstanding 
issues  to  discuss. 

The  JROC  and  its  supporting  organizations  review  requirements 
documents  related  to  capability  gaps  and  the  MDAPs  intended  to  fill  those 
gaps  prior  to  key  acquisition  milestones.  These  requirements  documents — 
the  Initial  Capabilities  Documents  (ICD),  Capability  Development 
Documents  (CDD),  and  Capability  Production  Documents  (CPD) — are 
submitted  by  capability  sponsors,  which  are  generally  the  military 
services,  but  can  also  be  other  DOD  agencies  or  combatant  commands. 
Figure  1  depicts  how  JCIDS  reviews  align  with  the  acquisition  process. 


uMajor  defense  acquisition  programs  are  those  programs  identified  by  DOD  that  require 
eventual  total  research,  development,  test,  and  evaluation  expenditures,  including  all 
planned  increments,  of  more  than  $365  million  or  procurement  expenditures,  including  all 
planned  increments,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant  dollars. 
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Figure  1 :  JCIDS  Reviews  and  the  DOD  Acquisition  Process 
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Source:  GAO  analysis  of  DOD  policy. 

Note:  Our  audit  primarily  focused  on  JROC  efforts  to  ensure  programs  were  initiated  on  a  sound 
basis,  i.e.,  requirements  validation  prior  to  program  start. 


The  ICD  is  the  first  requirements  document  reviewed  in  JCIDS.  It  is 
intended  to  identify  a  specific  capability  gap,  or  set  of  gaps,  in  joint 
military  capabilities  that  are  determined  to  require  a  materiel  solution  as  a 
result  of  a  capabilities-based  assessment.  DOD  policy  requires  that  the 
JROC  validate  the  ICD  prior  to  a  Materiel  Development  Decision,  which  is 
the  formal  entry  point  into  the  acquisition  process.  The  ICD  does  not 
contain  specific  cost,  schedule,  or  performance  objectives.  Once  the  JROC 
validates  an  ICD,  the  Milestone  Decision  Authority,  working  with 
appropriate  stakeholders,  shall  determine  whether  to  proceed  to  a 
Materiel  Development  Decision.12  After  the  Materiel  Development 
Decision,  the  capability  sponsor  initiates  an  AOA  to  consider  alternative 
solutions  to  fulfilling  the  capability  need  described  in  an  ICD,  and  possible 
trade-offs  among  cost,  schedule,  and  performance  for  each  alternative  are 
considered. 

The  CDD  is  the  second  requirements  document  reviewed  in  JCIDS.  It  can 
address  capability  gaps  presented  in  one  or  more  ICDs.  The  CDD  is 
intended  to  define  a  proposed  program’s  Key  Performance  Parameters 


12The  Milestone  Decision  Authority  is  an  acquisition  official  with  the  authority  to  approve  a 
program’s  entry  into  the  next  phase  of  the  acquisition  process.  For  MDAPs,  the  Milestone 
Decision  Authority  is  the  USD  AT&L,  DOD  component  head,  or  a  component  acquisition 
executive. 
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(KPP),  Key  System  Attributes  (KSA),  and  other  performance  attributes. 
KPPs  are  the  system  characteristics  that  the  CDD  sponsor  considers  most 
critical  to  delivering  that  military  capability,  while  KSAs  are  system 
attributes  the  CDD  sponsor  considers  essential  for  an  effective  military 
capability,  but  a  lower  priority  than  the  KPPs.  DOD  policy  calls  for  the 
JROC  to  validate  the  CDD  to  inform  the  Milestone  B  decision,  which 
marks  the  official  start  of  an  acquisition  program  and  entry  into  the 
engineering  and  manufacturing  development  phase.  The  CDD  is  the  first 
requirements  document  that  contains  cost,  schedule,  and  performance 
objectives. 

The  CPD  is  the  third  and  final  requirements  document  reviewed  in  JCIDS. 
It  is  intended  to  refine  the  KPPs,  KSAs,  and  performance  attributes 
validated  in  the  CDD.  DOD  policy  calls  for  the  JROC  to  validate  the  CPD  to 
inform  the  Milestone  C  decision,  which  marks  a  program’s  entry  into 
production.  Appendix  II  identifies  the  ICDs,  CDDs,  and  CPDs  reviewed  by 
the  JCB  or  JROC  in  fiscal  year  2010. 

In  addition  to  JCIDS  reviews,  the  JROC  reviews  MDAP  requirements  after 
a  program  experiences  cost  growth  beyond  JROC-specified  amounts, 
triggering  what  is  called  a  tripwire  review.  The  JROC’s  tripwire  policy 
directs  military  services  to  brief  the  JROC  when  unit  cost  increases  by  10 
percent  over  the  current  baseline  or  25  percent  over  the  original  baseline. 
DOD  officials  explained  this  allows  the  JROC  to  consider  relaxing, 
deferring,  or  deleting  a  program’s  KPPs  if  they  are  found  to  be 
unachievable  or  of  lesser  priority  than  reducing  program  cost.  The  JROC 
also  participates  in  what  are  commonly  known  as  critical  Nunn-McCurdy13 
reviews  by  providing  an  assessment  of  whether  the  program  is  essential  to 
national  security.  A  critical  Nunn-McCurdy  breach  occurs  when  the 
program  acquisition  unit  cost  or  the  average  procurement  unit  cost 
increases  by  at  least  25  percent  over  the  current  baseline  estimate  or  at 
least  50  percent  over  the  original  baseline  estimate.  Appendix  III  lists  the 
JROC’s  15  fiscal  year  2010  Nunn-McCurdy  and  tripwire  reviews. 


1310  U.S.C.  §  2433a. 
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JROC  Did  Not  Always 
Consider  Trade-offs 
or  Influence  Trade-off 
Decisions 


The  JROC  considered  trade-offs  made  by  the  military  services  before 
validating  requirements  for  four  of  the  seven  proposed  programs  it 
reviewed  in  fiscal  year  2010,  and  provided  input  to  the  military  services  on 
the  cost,  schedule,  and  performance  objectives  for  two  of  the  seven 
programs.  The  JROC’s  requirements  review  was  the  final  step  in  a  long 
requirements  vetting  process,  with  most  trade-offs  being  made  by  the 
military  services  earlier  in  the  process.  Key  stakeholders  from  the  offices 
of  the  Under  Secretary  of  Defense  (Comptroller),  USD  AT&L,  Director  of 
CAPE,  and  the  combatant  commands  were  all  satisfied  with  their 
opportunities  to  provide  input  to  the  JROC;  but  they  provided  limited 
input  on  trade-offs  among  cost,  schedule,  and  performance  objectives,  and 
used  other  means  to  influence  trade-offs.  Perhaps  most  importantly,  none 
of  the  JROC’s  requirements  reviews  align  with  the  AO  A,  which  is  where 
the  military  services  reported  making  the  most  significant  trade-offs.  As  a 
result,  a  program  can  spend  significant  time  in  technology  development 
before  the  JROC  gets  to  formally  weigh  in  on  these  trade-offs  through  the 
JCIDS  process.  The  JROC  also  reviews  MDAP  requirements  after  a 
program  enters  development  and  experiences  substantial  cost  growth. 
DOD  and  the  JROC  stated  that  requirements  were  not  the  primary  causes 
of  cost  growth  for  the  15  programs  reviewed  for  this  purpose  in  fiscal  year 
2010  and  the  JROC  did  not  change  any  KPPs  to  mitigate  the  reported  cost 
growth. 


JROC  Did  Not  Always 
Consider  Trade-offs  when 
Validating  Requirements 
for  Proposed  Programs 


The  JROC  considered  trade-offs  made  by  the  military  services  before 
validating  requirements  for  four  of  the  seven  proposed  programs  it 
reviewed  in  fiscal  year  2010.  On  three  programs,  the  JROC  did  not  receive 
information  on  the  potential  cost  and  schedule  implications  of  each  of  the 
alternatives  considered.  Table  1  summarizes  the  JROC’s  consideration  of 
cost,  schedule,  and  performance  objectives  for  the  seven  proposed  MDAPs 
it  reviewed  in  fiscal  year  2010. 


Page  8 


GAO-11-502  Missed  Trade-off  Opportunities 


Table  1 :  JROC  Consideration  of  Trade-offs  Among  Cost,  Schedule,  and  Performance  Objectives  for  Seven  Proposed 

Programs  in  Fiscal  Year  2010 

Proposed  program 

Capability  description 

Did  JROC  consider  cost,  schedule,  and 
performance  trade-offs? 

Common  Vertical  Lift  Support 
Platform  (CVLSP) 

The  CVLSP  program  is  expected  to  provide  a 
helicopter  that  will  support  nuclear  security  and 
passenger  transport  missions,  and  improve 
carrying  capacity,  range,  speed,  survivability,  and 
battlespace  awareness  among  other  capabilities. 

Yes.  The  JROC  established  rapid  fielding  as  the 
priority  and  instructed  the  Air  Force  to  discuss 
trade-offs  with  the  acquisition  and  testing 
communities  if  the  system’s  requirements  were 
adversely  affecting  its  rapid  fielding. 

HH-60  Recapitalization 

The  HH-60  Recapitalization  program  is  expected 
to  provide  a  helicopter  that  will  support  personnel 
recovery  missions,  and  be  capable  of  operating 
day  or  night,  in  adverse  weather,  and  amongst  a 
variety  of  threats. 

Yes.  The  JROC  approved  the  decision  to  change 
combat  radius,  survivability,  cabin  space, 
payload,  and  airspeed  requirements  to  decrease 
cost. 

Army  Integrated  Air  and 

Missile  Defense  System  of 
Systems  (AIAMD  SOS), 
Increment  2 

The  AIAMD  SOS,  Increment  2  program  is  expected 
to  provide  an  air  and  missile  defense  system  that 
will  integrate  sensors  and  weapons  across  an 
integrated  fire  control  network. 

Yes.  The  JROC  approved  the  decision  to  accept 
greater  cost  and  schedule  risk  in  order  to  meet 
performance  objectives. 

Ground  Soldier  System, 
Increment  1a 

The  Ground  Soldier  System,  Increment  1  program 
is  expected  to  provide  a  communications  system 
that  will  improve  ground  combat  leaders’  battle 
command  and  situational  awareness  capabilities. 

Yes.  The  JROC  emphasized  the  importance  of 
weight  and  mobility  and  directed  the  Army  to 
seek  JROC  approval  if  the  operating  weight 
exceeded  a  specific  threshold. 

Joint  Precision  Approach  and 
Landing  System  (JPALS), 
Increment  2 

The  JPALS,  Increment  2  program  is  expected  to 
provide  avionics  systems  and  mobile  and  fixed 
ground  stations  that  will  increase  the  efficiency  of 
approach  and  landing  operations. 

No.  The  JROC  received  brief  descriptions  of 
seven  potential  solutions,  but  the  cost  and 
schedule  implications  of  each  alternative  were 
not  identified. 

P-8A,  Increment  3b 

The  P-8A,  Increment  3  program  is  expected  to 
provide  an  antisurface  warfare  weapon  and 
improved  communications  capabilities. 

No.  The  JROC  received  brief  descriptions  of 
seven  analyses  supporting  the  P-8A  program,  but 
they  did  not  explicitly  discuss  Increment  3  cost, 
schedule,  and  performance  objectives. 

Ship  to  Shore  Connector 

The  Ship  to  Shore  Connector  program  is  expected 
to  provide  an  air  cushion  vehicle  for  transporting 
vehicles,  cargo,  and  personnel  from  ship  to  shore, 
and  will  replace  the  current  Landing  Craft  Air 
Cushion. 

No.  The  JROC  received  brief  descriptions  of 
three  potential  solutions,  but  the  cost  and 
schedule  implications  of  each  alternative  were 
not  identified. 

Source:  GAO  analysis  of  JCIDS  and  acquisition  documents. 


aThe  Ground  Soldier  System,  Increment  1  program  has  been  renamed  Nett  Warrior. 

bThe  proposed  P-8A,  Increment  3  program  has  not  yet  reached  Milestone  A,  but  at  the  direction  of  the 
Joint  Staff,  its  requirements  were  approved  in  the  same  document  as  the  proposed  P-8A,  Increment  2 
program. 


The  JROC’s  review  of  the  CDD  for  a  proposed  program  is  the  final  step  in 
a  long  requirements  vetting  process,  and  DOD  officials  reported  that  trade¬ 
offs  typically  occur  earlier  in  the  process.  Each  military  service  conducts 
its  own  internal  requirements  reviews  for  its  proposed  programs,  which 
are  used  to  refine  requirements  documents  before  they  are  submitted  into 
JCIDS.  Military  service  officials  reported  that  they  make  significant  trade- 
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offs  during  these  internal  reviews,  and  that  KPPs  and  technical 
requirements  rarely  change  after  requirements  documents  are  submitted 
into  JCIDS  because  extensive  analysis  has  already  been  conducted.  For 
the  seven  proposed  MDAPs  we  reviewed,  the  military  services  generally 
submitted  requirements  to  the  JROC  that  would  be  fully  funded,  provide 
initial  capability  within  6  years,  utilize  critical  technologies  that  were 
nearing  maturity,  and  be  acquired  using  an  incremental  approach.  These 
characteristics  are  consistent  with  provisions  in  the  Weapon  Systems 
Acquisition  Reform  Act  (WSARA)  related  to  how  the  requirements  process 
should  be  structured14  and  aspects  of  GAO’s  best  practices  for  weapon 
system  acquisitions.16 

Two  of  the  proposed  program  requirements  presented  to  the  JROC 
included  major  trade-offs  among  cost,  schedule,  and  performance 
objectives  and  revisions  to  their  acquisition  approaches  that  had  been 
made  after  predecessor  programs  were  cancelled  over  affordability 
concerns.  The  Air  Force  initiated  the  HH-60  Recapitalization  program  after 
the  Combat  Search  and  Rescue  Replacement  Vehicle  (CSAR-X)  program 
was  cancelled,  and  the  HH-60  Recapitalization  program  is  expected  to 
decrease  cost  by  changing  cabin  space,  velocity,  and  range  from  the  CSAR- 
X  requirements.  In  2007,  the  Army,  with  input  from  a  Functional 
Capabilities  Board,  decided  to  use  an  incremental  acquisition  approach  for 
the  Ground  Soldier  System  in  order  to  reduce  costs,  meet  schedule 
demands,  and  avoid  some  of  the  mistakes  made  during  the  Land  Warrior 
program,  which  was  cancelled  because  of  funding  and  cost  issues. 


Key  Stakeholders  Provided 
Limited  Input  into  JCIDS, 
but  Use  Other  Means  to 
Influence  Trade-offs 


The  JROC  received  limit  ed  input  on  trade-offs  among  cost,  schedule,  and 
performance  objectives  from  key  stakeholders  when  validating 
requirements  for  the  seven  proposed  MDAPs  we  reviewed  from  fiscal  year 
2010.  Both  WSARA  and  the  National  Defense  Authorization  Act  for  Fiscal 
Year  2008  directed  the  JROC  to  consult  with  the  Under  Secretary  of 


14Pub.  L.  No.  111-23  §  201(a)(2)(B). 

15GAO,  Defense  Acquisitions:  A  Knowledge-Based  Funding  Approach  Could  Improve 
Major  Weapon  System.  Program.  Outcomes,  GAO-08-619  (Washington,  D.C.:  July  2008). 
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Defense  (Comptroller),  the  USD  AT&L,  and  the  Director  of  CAPE.16 
Additionally,  WSARA  instructed  the  JROC  to  consult  with  the  combatant 
commands.  Officials  from  these  organizations  reported  that  they  had 
ample  opportunity  to  participate  in  JROC  requirements  reviews,  and  Joint 
Staff  officials  said  efforts  to  involve  these  stakeholders  preceded  WSARA. 
However,  officials  from  the  offices  of  the  Under  Secretary  of  Defense 
(Comptroller),  USD  AT&L,  and  the  Director  of  CAPE  also  reported  that 
the  acquisition  and  budgeting/funding  processes  are  the  primary 
mechanisms  through  which  they  influence  programs,  rather  than  JCIDS. 
For  example,  CAPE  oversees  AOAs  for  MDAPs  and  has  an  opportunity  to 
provide  input  and  guidance  on  AOA  considerations.  Further,  the 
combatant  commands  reported  that  they  most  often  submit  prioritized 
lists  of  capability  gaps  directly  to  the  Chairman  of  the  Joint  Chiefs  of  Staff 
as  part  of  the  resource  allocation  process,  which  is  separate  from  JCIDS.17 

Nonetheless,  joint  stakeholders  did  provide  some  significant  input  during 
the  JROC’s  reviews  of  the  seven  proposed  programs  in  fiscal  year  2010. 
For  example,  the  Army  more  fully  defined  a  Ground  Soldier  System, 
Increment  1  KPP  in  response  to  input  from  DOD’s  Joint  Interoperability 
Test  Command,  and  in  another  instance,  the  Army  added  a  KSA  to  the 
ALAMD  SOS,  Increment  2  CDD  due  to  input  from  the  office  of  the  USD 
AT&L,  the  Defense  Information  Systems  Agency,  and  the  Joint  Staff. 
Neither  of  these  changes  involved  trade-offs  among  cost,  schedule,  and 
performance  objectives. 


JCIDS  Reviews  Are  Not 
Aligned  with  the  Most 
Significant  Trade-off 
Decisions 


The  JROC  does  not  formally  review  the  trade-off  decisions  made  as  a 
result  of  an  AOA  until  a  proposed  program’s  CDD  enters  the  JCIDS 
process.  According  to  DOD  officials,  the  most  significant  trade-offs  are 
made  by  the  military  services  between  ICD  and  CDD  reviews  during  the 
AOA,  which  is  intended  to  compare  the  operational  effectiveness,  cost, 
and  risks  of  a  number  of  alternative  potential  solutions.  For  example, 
during  the  CVLSP  AOA,  the  Air  Force  decided  to  decrease  troop  transport 


“Prior  to  WSARA,  the  Director  of  the  Office  of  Program  Analysis  and  Evaluation,  which  is 
named  as  a  JROC  advisor  in  the  National  Defense  Authorization  Act  of  Fiscal  Year  2008, 
performed  some  of  the  cost  estimating  and  program  evaluation  roles  now  performed  by 
CAPE.  The  Deputy  Director  of  Cost  Assessment  performs  independent  cost  estimates  for 
MDAPs,  among  other  roles.  The  Deputy  Director  of  Program  Evaluation  performs 
evaluations  of  the  net  costs  and  benefits  of  a  proposed  system  compared  with  alternatives, 
among  other  roles. 

17GAO-ll-527R. 
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capacity  in  order  to  reduce  cost.  Alternatively,  during  the  AIAMD  SOS 
AO  A,  the  Army  decided  to  pursue  the  most  costly  option  reviewed 
because  it  provided  greater  capability.  A  significant  amount  of  time  and 
resources  can  be  expended  before  the  JROC  gets  to  weigh  in  on  these 
trade-offs  during  ODD  reviews.  For  example,  the  JROC  reviewed  the  AOA 
summary  for  JPALS,  Increment  2,  4  years  after  the  conclusion  of  the  AOA. 
During  the  time  between  the  AOA  and  the  CDD  review,  the  technology 
intended  to  enable  the  chosen  alternative  is  developed.  Figure  2  shows  the 
AOA’s  relationship  to  both  the  requirements  and  acquisition  processes. 


Figure  2:  AOA’s  Relationship  to  JROC  Reviews 


Joint  Staff  officials  have  stated  that  establishing  a  JROC  review  of  the  AOA 
would  allow  it  to  provide  military  advice  on  trade-offs  and  the  proposed 
materiel  solution  before  Milestone  A,  and  an  ongoing  Joint  Staff  review  of 
JCIDS  is  considering  an  increased  role  for  the  JROC  at  this  point.18 
According  to  the  Joint  Staff,  increased  JROC  engagement  at  these  early 
stages  of  the  acquisition  process  is  warranted  to  align  it  with  other 


lsThe  Joint  Capabilities  Development  Process  Review  Integrated  Process  Team  is  providing 
recommendations  to  improve  JCIDS’  responsiveness  and  decision  support  to  the  JROC, 
combatant  commands,  military  services,  and  defense  agencies.  The  target  completion  date 
for  implementing  the  review’s  recommendations  is  June  30,  2011. 
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elements  of  recent  acquisition  reforms.  For  example,  WSARA  emphasized 
that  the  AOA  should  fully  consider  possible  trade-offs  among  cost, 
schedule,  and  performance  objectives  for  each  alternative  considered,19 
and  in  September  2010,  USD  AT&L  issued  a  memorandum  that 
emphasized  the  need  for  trade-offs  from  a  program’s  inception.20  The 
memorandum  also  dictated  that  affordability  targets  shall  be  established  at 
the  conclusion  of  the  AOA  and  that  these  targets  will  be  treated  like  KPPs, 
even  though  they  will  be  set  and  managed  by  the  acquisition,  not 
requirements,  community. 


JROC  Did  Not  Change 
KPPs  when  Programs 
Incurred  Substantial  Cost 
Growth 


The  JROC  did  not  change  any  KPPs  during  15  reviews  of  programs  that 
reported  substantial  cost  growth  in  fiscal  year  2010.  According  to  the  Joint 
Staff,  by  holding  requirements  firm  and  accepting  increased  cost  and 
schedule  delays,  the  JROC  essentially  traded  cost  and  possibly  schedule 
for  performance.  In  fiscal  year  2010,  the  JROC  reviewed  six  programs  after 
they  experienced  a  critical  Nunn-McCurdy  breach  and  nine  programs  as 
part  of  the  tripwire  process.  During  all  15  reviews,  DOD  and  the  JROC 
stated  that  requirements  were  not  the  primary  causes  of  cost  growth.  For 
all  six  programs  that  experienced  a  critical  Nunn-McCurdy  cost  breach, 
the  JROC  validated  the  system’s  capabilities  as  being  essential  to  national 
security  and  did  not  make  any  changes  to  their  KPPs.  For  all  nine 
programs  that  were  approaching  Nunn-McCurdy  thresholds,  the  JROC  did 
not  identify  opportunities  to  mitigate  cost  growth  by  modifying 
requirements.  Most  of  these  programs  were  in  production  in  fiscal  year 
2010,  and  changing  requirements  at  this  late  stage  might  not  have 
mitigated  the  reported  cost  growth.  When  the  JROC  reviewed  the  Family 
of  Advanced  Beyond  Line-of-Sight  Terminals  program,  which  was  still  in 
development,  it  concluded  that  the  program’s  requirements  could  not  be 
met  in  an  affordable  manner.  The  JROC  did  not  immediately  defer  any  of 
the  program’s  requirements,  but  instead  requested  that  USD  AT&L  identify 
potential  alternatives  for  the  program,  including  reviewing  whether 
adjustments  to  performance  requirements  would  be  appropriate. 


19Pub.  L.  No.  111-23  §  201(d)(1). 

20Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  Better  Buy  i  ng 
Power:  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in  Defense 
Spending  (Washington,  D.C.:  Sept.  14,  2010). 
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Military  Services  Did 
Not  Consistently 
Provide  High-Quality 
Resource  Estimates 
to  JROC 


The  military  services  did  not  consistently  provide  high-quality  resource 
estimates  to  the  JROC  to  support  its  review  of  requirements  for  7 
proposed  programs  in  fiscal  year  2010.  We  found  the  estimates  presented 
to  the  JROC  were  often  unreliable  when  assessed  against  best  practices 
criteria.  The  type  of  resource  estimates  the  military  services  presented  to 
the  JROC  varied  from  ones  that  had  been  validated  by  the  military 
services’  cost  analysis  agencies  to  less  rigorous  rough-orders-of-magnitude 
estimates.  In  most  cases,  the  military  services  had  not  effectively 
conducted  uncertainty  and  sensitivity  analyses,  which  establish 
confidence  levels  for  resource  estimates,  based  on  the  knowledge 
available,  and  examine  the  effects  of  changing  assumptions  and  ground 
rules.  Lacking  risk  and  uncertainty  analysis,  the  JROC  cannot  evaluate  the 
range  of  resources  that  might  be  necessary  to  cover  increased  costs 
resulting  from  unexpected  design  complexity,  technology  uncertainty,  and 
other  issues.  The  lack  of  this  information  affects  the  JROC’s  efforts  to 
ensure  that  programs  are  fully  funded  and  its  ability  to  consider  the 
resource  implications  of  cost,  schedule,  and  performance  trade-offs. 


JROC  Received  Resource  The  JROC  first  receives  resource  estimates  for  proposed  programs  when  it 
Estimates  That  Did  Not  reviews  CDDs,  and  when  we  reviewed  the  CDD  resource  estimates 

Meet  Best  Practices  presented  to  the  JROC  in  fiscal  year  2010,  we  found  that  they  were 

generally  unreliable  when  assessed  against  our  best  practices  criteria.21 
While  most  of  the  resource  estimates  substantially  met  our  criteria  for  a 
comprehensive  resource  estimate,  they  generally  were  not  very  accurate, 
credible,  or  well-documented.  Appendix  IV  includes  a  list  of  the  best 
practices  against  which  we  assessed  these  resource  estimates. 

The  type  of  resource  estimates  the  military  services  presented  to  the  JROC 
varied  from  ones  that  had  been  validated  by  the  military  services’  cost 
analysis  agencies  to  less  rigorous  rough-orders-of-magnitude  estimates.22 
According  to  Joint  Staff  officials,  military  services  can  initiate  CDD 
reviews  at  any  point  in  the  acquisition  process  prior  to  program  start,  even 
if  good  resource  estimates  are  not  available.  For  example,  the  JROC 
validated  the  P-8A,  Increment  3  CDD  more  than  2  years  before  the 
program  was  expected  to  start,  before  an  AOA  had  been  completed,  and 
with  a  rough-order-of-magnitude  estimate.  Joint  Staff  officials  reported 


Z1GAO-09-3SP. 

22A  rough-order-of-magnitude  estimate  is  a  quick,  high-level  estimate  that  generally  involves 
less  time  and  effort  than  a  budget-quality  estimate. 
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that  they  depend  on  CAPE  to  review  the  quality  of  resource  estimates 
during  the  JCIDS  process,  but  CAPE  cost  assessment  officials  told  us  that 
they  rarely  participate  in  JCIDS  reviews. 

Regardless  of  the  type  of  resource  estimate,  uncertainty  and  sensitivity 
analysis  can  establish  confidence  levels  for  resource  estimates,  based  on 
the  knowledge  available  at  the  time,  and  examine  the  effects  of  changing 
assumptions  and  ground  rules,  including  those  related  to  trade-offs  among 
cost,  schedule,  and  performance  objectives.  The  military  services 
sponsoring  the  requirements  generally  did  not  effectively  meet  best 
practices  for  uncertainty  and  sensitivity  analyses  using  the  knowledge  they 
had  available  to  them  for  any  of  the  seven  resource  estimates  we 
reviewed.  Figure  3  summarizes  our  assessment  of  the  resource  estimates 
presented  to  the  JROC  against  our  best  practices  criteria. 


Figure  3:  GAO  Assessment  of  Resource  Estimates  Presented  to  the  JROC  During  CDD  Reviews  in  Fiscal  Year  2010 


CDDs 

Type  of  resource  estimate 

Characteristics  of  a  reliable  resource  estimate 

Comprehensive 

Accurate 

Credible 

Well-documented 

Ship  to  Shore  Connector 

Naval  Center  for  Cost  Analysis 
validated  estimate 

© 

© 

© 

JPALS,  Increment  2 

Air  Force  cost  estimate 

© 

© 

© 

© 

CVLSP 

Air  Force  Cost  Analysis  Agency 
validated  estimate3 

© 

© 

© 

© 

AIAMD  SOS,  Increment  2 

Army  cost  position 

© 

© 

© 

© 

Ground  Soldier  System,  Increment  1 

Program  office  estimate 

© 

© 

© 

© 

HH-60  Recapitalization 

Rough-order-of-magnitude  estimate 

© 

© 

o 

© 

P-8,  Increment  3 

Rough-order-of-magnitude  estimate 

© 

o 

o 

o 

^  Met  -  evidence  satisfied  the  entire  criterion 
^  Substantially  met  -  evidence  satisfied  a  large  portion  of  the  criterion 
^  Partially  met  -  evidence  satisfied  about  half  of  the  criterion 
©  Minimally  met  -  evidence  satisfied  a  small  portion  of  the  criterion 
o  Not  met  -  no  evidence  satisfied  any  of  the  criterion 
Source:  GAO  analysis  of  DOD  data. 

“Air  Force  Cost  Analysis  Agency  found  the  CVLSP  estimate  adequate  for  an  AOA,  but  not  for 
budgeting  purposes. 
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Most  Resource  Estimates 
Substantially  Met 
Comprehensiveness  Criteria 


Most  Resource  Estimates  Did 
Not  Substantially  Meet 
Accuracy  Criteria 


Five  of  the  seven  CDD  resource  estimates  substantially  met  our  criteria  for 
a  comprehensive  resource  estimate.  The  resource  estimates  generally 
completely  defined  their  respective  programs,  and  included  most,  if  not 
all,  life-cycle  costs.  The  Ship  to  Shore  Connector,  CVLSP,  and  JPALS, 
Increment  2  resource  estimates  also  effectively  documented  all  cost- 
influencing  ground  rules  and  assumptions,  although  the  other  resource 
estimates  did  not.  Additionally,  only  the  Ship  to  Shore  Connector’s  work 
breakdown  structure  effectively  met  our  criteria,  which  require  that  work 
breakdown  structures  are  product-oriented  and  at  an  appropriate  level  of 
detail.  If  a  resource  estimate  does  not  specifically  break  out  common 
costs,  such  as  government-furnished  equipment  costs,  or  does  not  include 
an  associated  work  breakdown  structure  dictionary,  cost  estimators 
cannot  ensure  that  the  estimate  includes  all  relevant  costs. 

The  HH-60  Recapitalization  and  P-8A,  Increment  3  resource  estimates  did 
not  effectively  meet  any  of  our  best  practices  for  a  comprehensive 
resource  estimate.  Unless  resource  estimates  account  for  all  costs,  they 
cannot  enhance  decision  making  by  allowing  for  design  trade-off  studies 
to  be  evaluated  on  a  total  cost,  technical,  and  performance  basis. 
Additionally,  unless  ground  rules  and  assumptions  are  clearly 
documented,  the  resource  estimate  will  not  have  a  basis  for  resolving 
areas  of  potential  risk. 

Only  two  of  the  seven  CDD  resource  estimates  substantially  met  our 
criteria  for  an  accurate  resource  estimate,  while  three  partially  met  the 
criteria,  and  two  did  not  meet  or  minimally  met  the  criteria.  We  found  that 
the  Ship  to  Shore  Connector,  CVLSP,  AIAMD  SOS,  Increment  2,  and  the 
Ground  Soldier  System,  Increment  1  resource  estimates  contained  few,  if 
any,  minor  mistakes,  and  that  the  Ship  to  Shore  Connector,  CVLSP,  and 
JPALS,  Increment  2  resource  estimates  were  appropriately  adjusted  for 
inflation.  Additionally,  we  found  that  the  Ship  to  Shore  Connector  and 
JPALS,  Increment  2  resource  estimates  were  based  on  historical  records 
of  actual  experiences  from  other  comparable  programs. 

However,  we  generally  found  that  the  resource  estimates  were  not 
consistent  with  our  best  practices.  Accurate  resource  estimates  are  rooted 
in  historical  data,  which  provide  cost  estimators  with  insight  into  actual 
costs  of  similar  programs,  and  can  be  used  to  challenge  optimistic 
assumptions  and  bring  more  realism  to  a  resource  estimate.  Unless  an 
estimate  is  based  on  an  assessment  of  the  most  likely  costs,  and  reflects 
the  degree  of  uncertainty  given  all  of  the  risks  considered,  management 
will  not  be  able  to  make  well-informed  decisions. 
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Most  Resource  Estimates  Were 
Not  Credible 


Four  of  the  seven  CDD  resource  estimates  did  not  meet  or  minimally  met 
our  criteria  for  a  credible  resource  estimate,  and  only  the  Ship  to  Shore 
Connector  resource  estimate  substantially  met  the  criteria.  The  Ship  to 
Shore  Connector  and  AIAMD  SOS,  Increment  2  resource  estimates 
included  sensitivity  analyses  that  identified  a  range  of  possible  costs  based 
on  varying  assumptions,  parameters,  and  data  inputs,  but  none  of  the 
other  resource  estimates  included  this  analysis.  As  a  best  practice, 
sensitivity  analysis  should  be  included  in  all  resource  estimates  because  it 
examines  the  effects  of  changing  assumptions  and  ground  rules.  Since 
uncertainty  cannot  be  avoided,  it  is  necessary  to  identify  the  cost  elements 
that  represent  the  most  risk  and,  if  possible,  cost  estimators  should 
quantify  that  risk.  When  an  agency  fails  to  conduct  sensitivity  analysis  to 
identify  the  effect  of  uncertainties  associated  with  different  assumptions, 
this  increases  the  chance  that  decisions  will  be  made  without  a  clear 
understanding  of  the  impact  on  cost. 

Additionally,  only  the  Ship  to  Shore  Connector  resource  estimate 
effectively  met  our  best  practices  for  risk  and  uncertainty  analysis.  For 
management  to  make  good  decisions,  the  program  estimate  must  reflect 
the  degree  of  uncertainty  so  that  a  level  of  confidence  can  be  given  about 
the  estimate.  An  estimate  without  risk  and  uncertainty  analysis  is 
unrealistic  because  it  does  not  assess  the  variability  in  the  resource 
estimate  from  effects  such  as  schedules  slipping,  missions  changing,  and 
proposed  solutions  not  meeting  users’  needs.  Lacking  risk  and  uncertainty 
analysis,  management  cannot  determine  a  defensible  level  of  contingency 
reserves  that  is  necessary  to  cover  increased  costs  resulting  from 
unexpected  design  complexity,  technology  uncertainty,  and  other  issues. 

Further,  none  of  the  planned  programs  effectively  met  our  criteria  for  an 
independent  cost  estimate  when  they  were  reviewed  by  the  JROC.  An 
independent  cost  estimate  is  considered  one  of  the  best  and  most  reliable 
resource  estimate  validation  methods  because  it  provides  an  independent 
view  of  expected  program  costs  that  tests  the  program  office  and  service 
estimates  for  reasonableness.  Without  an  independent  cost  estimate, 
decision  makers  lack  insight  into  a  program’s  potential  costs  because 
these  estimates  frequently  use  different  methods  and  are  less  burdened 
with  organizational  bias.  Moreover,  independent  cost  estimates  tend  to 
incorporate  adequate  risk,  and  therefore  tend  to  be  more  conservative  by 
forecasting  higher  costs  than  the  program  office.  A  program  estimate  that 
has  not  been  reconciled  with  an  independent  cost  estimate  has  an 
increased  risk  of  proceeding  underfunded  because  an  independent  cost 
estimate  provides  an  objective  and  unbiased  assessment  of  whether  the 
program  estimate  can  be  achieved.  Alternatively,  programs  can  reinforce 
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the  credibility  of  their  resource  estimates  through  cross-checking,  which 
determines  whether  alternative  cost  estimating  methods  produce  similar 
results.  However,  only  the  Ship  to  Shore  Connector  resource  estimate 
effectively  met  our  best  practices  for  cross-checking. 

Most  Resource  Estimates  Were  Only  the  JPALS,  Increment  2  resource  estimate  substantially  met  our 

Not  Well-documented  criteria  for  a  well-documented  resource  estimate,  while  four  of  the  seven 

CDD  resource  estimates  partially  met  our  criteria,  and  two  of  the  resource 
estimates  did  not  meet  or  minimally  met  the  criteria.  The  JPALS, 

Increment  2  and  CVLSP  resource  estimates  sufficiently  described  the 
calculations  performed  and  estimating  methodologies  used  to  derive  each 
program  element’s  cost.  Additionally,  the  JPALS,  Increment  2,  Ship  to 
Shore  Connector,  and  AIAMD  SOS,  Increment  2  documentation  clearly 
discusses  the  technical  baseline  description,  and  the  data  in  the  technical 
baseline  are  consistent  with  the  resource  estimate.  However,  none  of  the 
documents  effectively  described  how  the  resource  estimates  were 
developed  in  a  manner  that  a  cost  analyst  unfamiliar  with  the  program 
could  understand  what  was  done  and  replicate  it. 

We  generally  found  that  the  resource  estimates  were  not  consistent  with 
our  best  practices  for  a  well-documented  resource  estimate. 
Documentation  is  essential  for  validating  and  defending  a  resource 
estimate.  Without  a  well-documented  resource  estimate,  a  convincing 
argument  of  an  estimate’s  validity  cannot  be  presented,  and  decision 
makers’  questions  cannot  be  effectively  answered.  Poorly  documented 
resource  estimates  cannot  explain  the  rationale  of  the  methodology  or  the 
calculations  underlying  the  cost  elements.  Further,  a  well-documented 
resource  estimate  is  essential  for  an  effective  independent  review  to 
ensure  that  the  resource  estimate  is  valid  and  credible.  Unless  the  estimate 
is  fully  documented,  it  will  not  support  reconciliation  with  an  independent 
cost  estimate,  hindering  understanding  of  cost  elements  and  their 
differences. 


Military  Services  Generally 
Presented  Resource 
Estimates  That  Were  Fully 
Funded  to  JROC 


The  JROC  required  the  military  services  to  show  that  the  proposed 
programs  were  fully  funded  to  the  resource  estimates  presented  by  the 
military  services  before  it  validated  requirements  for  five  of  the  seven 
proposed  MDAPs  we  reviewed  from  fiscal  year  2010;  the  two  other 
proposed  MDAPs  were  funded  at  more  than  97  and  99  percent 
respectively.23  However,  we  found  that  these  resource  estimates  were 


23Full  funding  refers  to  a  budgetary  allocation  in  the  future-years  defense  program. 
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generally  unreliable,  which  undermined  the  JROC’s  efforts.  In  2007,  the 
JROC  issued  guidance  instructing  the  military  services  to  commit  to 
funding  the  requirements  that  the  JROC  validates.  The  guidance 
emphasized  the  need  for  full  funding  in  an  effort  to  facilitate  sound  fiscal 
and  risk  decisions.24  However,  the  JROC  does  not  explicitly  consider  a 
requirement’s  affordability  in  a  broader  context  during  JCIDS  reviews. 
DOD  funding  plans  are  captured  in  the  future-years  defense  program, 
which  presents  resource  information  for  the  current  year  and  the 
following  4  years.  The  future-years  defense  program  is  updated  twice  per 
year  to  reflect  the  military  services’  input  and  the  budget  the  President 
submits  to  Congress.  Statute  and  DOD  acquisition  policy  also  require 
programs  to  be  fully  funded  through  the  period  covered  by  the  future- 
years  defense  program.26 

One  of  the  seven  proposed  MDAPs  we  reviewed  from  fiscal  year  2010 
included  a  funding  shortfall  when  its  requirements  were  being  reviewed 
through  JCIDS,  but  its  CDD  was  not  approved  until  the  shortfall  had  been 
addressed.  Specifically,  when  the  JCB  reviewed  the  CVLSP  CDD,  the 
funding  plan  included  a  $1.3  billion  shortfall  through  fiscal  year  2015.  The 
JCB  chairman  directed  the  Air  Force  to  modify  the  program’s  funding  plan 
before  proceeding  to  the  JROC  review.  When  the  Air  Force  briefed  the 
JROC  on  the  CVLSP  CDD  approximately  8  months  later,  it  presented  a 
funding  plan  that  fully  funded  the  program  through  the  future-years 
defense  program  time  frame.  The  revised  funding  plan  also  included  more 
money  for  the  program  beyond  the  future-years  defense  program  time 
frame,  and  the  total  program  cost  increased  from  $14.2  billion  to  $15.2 
billion. 

Despite  JROC  efforts  to  ensure  programs  are  fully  funded,  the  military 
services  retain  primary  control  over  their  budgets,  and  ultimately,  JROC 
decisions  are  influential  but  not  binding.  When  the  JCB  reviewed  the 
JPALS,  Increment  2  CDD,  it  requested  clarification  on  the  Air  Force’s 
funding  plan,  and  emphasized  the  need  for  full  funding  prior  to  program 
start.  The  funding  plan  presented  to  the  JCB  included  a  $77.7  million 


"4JROC,  Funding  Guidance  for  Joint  Requirements  Oversight  Council  Directed  Action, 
JROC  Memorandum  067-07  (Washington,  D.C.:  Mar.  23,  2007). 

J  'Ari  MDAP  may  also  not  receive  Milestone  B  approval  until  the  Milestone  Decision 
Authority  certifies  that  funding  is  available  to  execute  the  program  through  the  period 
covered  by  the  future-years  defense  program.  10  U.S.C.  §  2366b(a)(l)(D).  DOD  Instruction 
5000.02,  Operation  of  the  Defense  Acquisition  System  (Dec.  8,  2008)  also  requires  programs 
to  be  fully  funded  through  the  future-years  defense  program. 
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shortfall  through  fiscal  year  2015,  and  the  Air  Force  had  cut  JPALS  funding 
in  the  past.  Following  the  JCB  review,  the  JROC  issued  a  decision 
memorandum  that  documented  the  Air  Force’s  commitment  to  fully 
funding  JPALS,  Increment  2.  However,  in  fiscal  years  2011  and  2012,  the 
Air  Force  only  funded  approximately  30  percent  of  the  resource  estimate 
presented  to  the  JCB. 


JROC  Did  Not 
Consistently  Prioritize 
Requirements  and 
Capability  Gaps 


The  JROC  does  not  currently  prioritize  requirements,  consider 
redundancies  across  proposed  programs,  or  prioritize  and  analyze 
capability  gaps  in  a  consistent  manner.  As  a  result,  the  Joint  Staff  is 
missing  an  opportunity  to  improve  military  service  and  departmentwide 
portfolio  management  efforts.  A  portfolio  management  approach  to 
weapon  system  investments  would  involve  taking  a  disciplined,  integrated 
approach  to  prioritizing  needs  and  allocating  resources  in  order  to 
eliminate  redundancies,  gain  efficiencies,  and  achieve  a  balanced  mix  of 
executable  programs.  According  to  Army,  Air  Force,  and  Navy  officials, 
having  a  better  understanding  of  warfighter  priorities  from  the  JROC 
would  be  useful  to  inform  both  portfolio  management  efforts  and  service 
budgets.  A  DOD  review  team  examining  the  JCIDS  process  is  considering 
changes  that  would  address  the  prioritization  of  requirements.  During  its 
review  of  the  capability  gaps  presented  in  12  ICDs  in  fiscal  year  2010,  the 
JROC  did  receive  some  information  on  priorities  and  potential 
redundancies;  however,  the  sponsors  presented  this  information  in  an 
inconsistent  manner,  making  it  difficult  for  the  JROC  to  assess  the  relative 
priority  of  capability  gaps  across  different  ICDs. 


JROC  Does  Not  Prioritize 
Requirements  or  Consider 
Redundancies  Across 
Proposed  Programs 


Under  the  current  JCIDS  process,  the  JROC  does  not  prioritize 
requirements  or  consider  redundancies  across  proposed  programs  during 
CDD  reviews.  In  the  National  Defense  Authorization  Act  for  Fiscal  Year 
2008,  Congress  amended  the  U.S.  Code  to  direct  the  JROC  to  help  assign 
priority  levels  for  joint  military  requirements  and  ensure  that  resource 
levels  associated  with  those  requirements  are  consistent  with  the  level  of 
priority.26  The  House  Armed  Services  Committee  report  accompanying  the 
authorization  act  stated  that  clear  JROC  priorities  and  budget  guidance 
would  allow  for  joint  decision  making,  as  opposed  to  service-centric 
budget  considerations.27  In  addition,  we  have  previously  recommended 


26Pub.  L.  No.  110-181,  §  942,  (codified  at  10  U.S.C.  §  181(b)). 
27H.R.  Rep.  No.  110-146  at  381  (2007). 
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that  DOD  develop  an  analytic  approach  within  JCIDS  to  better  prioritize 
and  balance  the  capability  needs  of  the  military  services,  combatant 
commands,  and  other  defense  components.28  According  to  the  Joint  Staff 
and  military  service  officials,  prioritization  across  programs  still  primarily 
occurs  through  the  Planning,  Programming,  Budgeting  and  Execution 
process,  which  is  the  responsibility  of  the  military  services  and  the  Office 
of  the  Under  Secretary  of  Defense  (Comptroller). 

The  JCIDS  manual  does  not  currently  require  an  analysis  of  potential 
redundancies  during  CDD  reviews.  In  our  recently  issued  report  on 
government  duplication,  we  noted  that  service-driven  requirements  and 
funding  processes  continue  to  hinder  integration  and  efficiency  and 
contribute  to  unnecessary  duplication  in  addressing  warfighter  needs.28  We 
have  also  previously  reported  that  ineffective  collaboration  precluded 
opportunities  for  commonality  in  unmanned  aircraft  systems.30  In  fiscal 
year  2010,  the  JROC  met  to  consider  joint  efficiencies  between  two  such 
systems:  the  Navy’s  Broad  Area  Maritime  Surveillance  system  and  the  Air 
Force’s  Global  Hawk  system.  The  JROC  requested  that  the  Navy  and  Air 
Force  ensure  that  a  common  component  was  interoperable  between  the 
two  systems,  and  that  the  Air  Force  consider  an  all-weather  capability 
developed  by  the  Navy.  The  JROC  has  also  supported  joint  development 
efforts  for  these  programs  and  requested  annual  status  updates.  According 
to  Broad  Area  Maritime  Surveillance  program  officials,  the  Air  Force  and 
Navy  programs  are  investigating  commonality  opportunities,  including 
sense-and-avoid  capabilities,  a  consolidated  maintenance  hub,  and  basing 
options  for  both  systems.  The  JROC  did  not  meet  to  consider  any  other 
joint  efficiencies  across  military  services  in  fiscal  year  2010. 

The  Joint  Staff  has  acknowledged  that  the  JROC  should  play  a  larger  role 
in  prioritizing  needs  and  addressing  redundancies.  In  July  2010,  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  initiated  a  review  of  the  JCIDS 
process.  One  of  the  goals  of  the  review  team  was  to  develop  metrics  and 


28GAO,  Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been 
Effective  in  Prioritizing  Joint  Capabilities,  GAO-08-1060  (Washington,  D.C.:  September 
2008). 

29GAO,  Opportunities  to  Reduce  Potential  Duplication  in  Government  Programs,  Save 
Tax  Dollars,  and  Eyihance  Revenue,  GAO-11-318SP  (Washington,  D.C.:  March  2011). 

30GAO,  Defense  Acquisitions:  Opportunities  Exist  to  Achieve  Greater  Commonality  and 
Efficiencies  among  Unmanned  Aircraft  Systems,  GAO-09-520  (Washington,  D.C.:  Joly 
2009). 
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criteria  to  ensure  the  JCIDS  process  has  the  ability  to  rank  or  prioritize 
needs.  The  review  team’s  charter  states  that  these  metrics  must  enable 
more  structured  reviews  of  portfolio  gaps  and  redundancies.  According  to 
the  Joint  Staff,  the  review  team  is  considering  a  number  of 
recommendations  including  asking  the  JROC  to  prioritize  requirements 
based  on  the  urgency  and  significance  of  the  need.  This  list  of  priorities 
could  be  used  to  inform  military  service  budgets.  Joint  Staff  officials  have 
also  stated  that  redundancies  may  be  addressed  more  directly  in  the  future 
as  part  of  an  enhanced  portfolio  management  effort. 


Lack  of  JROC 
Prioritization  Results  in 
Missed  Opportunities  to 
Manage  Portfolios  Better 


We  have  previously  reported  that  DOD  has  not  taken  a  portfolio 
management  approach  to  weapon  system  investments,  which  would 
involve  taking  a  disciplined,  integrated  approach  to  prioritizing  needs  and 
allocating  resources  in  order  to  eliminate  redundancies,  gain  efficiencies, 
and  achieve  a  balanced  mix  of  executable  programs.31  In  September  2010, 
USD  AT&L  issued  guidance  intended  to  increase  efficiencies  and  eliminate 
redundancies,  and  it  presented  the  Army’s  portfolio  management  activities 
as  an  example  to  emulate.32  The  Army  uses  capability  portfolio  reviews  of 
capability  gaps  and  proposed  and  existing  programs  to  revalidate,  modify, 
or  terminate  requirements  and  ensure  the  proper  allocation  of  funds 
between  them.  The  Army  has  established  17  portfolios,  including  aviation, 
air  and  missile  defense,  and  combat  vehicle  modernization.  An  Army 
official  involved  in  the  portfolio  reviews  said  that  he  has  requested  on 
several  occasions  for  the  Joint  Staff  to  prioritize  warfighter  needs; 
however,  the  JROC  has  not  done  so.  Instead,  the  Army  relies  on  its  own 
prioritization  information  during  the  portfolio  reviews  to  help  determine 
the  capability  areas  where  it  is  willing  to  assume  risk.  Air  Force  and  Navy 
officials  have  also  stated  that  they  could  benefit  from  JROC  prioritization 
of  requirements,  and  that  this  information  would  be  useful  in  order  to 
better  allocate  resources  during  their  budget  formulation  activities. 


31GAO-07-388. 

32Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  Better  Buying 
Power':  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in  Defense 
Spending  (Washington,  D.C.:  Sept.  14,  2010). 
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Sponsors  Do  Not  Prioritize 
Capability  Gaps  or  Analyze 
Potential  Redundancies  in 
a  Consistent  Manner 


The  JROC  has  required  that  capability  sponsors  prioritize  capability  gaps 
and  identify  redundancies  when  developing  ICDs,33  and  capability 
sponsors  generally  complied  with  these  requirements  in  the  12  validated 
ICDs  we  reviewed  from  fiscal  year  20 10.34  However,  the  sponsors 
presented  this  information  in  an  inconsistent  manner,  making  it  difficult 
for  the  JROC  and  the  military  services  to  assess  priorities  and 
redundancies  across  ICDs  or  use  this  information  to  inform  resource 
allocation  decisions.  For  example,  the  Electronic  Health  Record  ICD 
prioritized  its  gaps  in  numerical  order  from  1  to  10, 36  but  the  Command  and 
Control  On-The-Move  ICD  labeled  half  its  gaps  medium  priority  and  the 
other  half  high  priority.36  The  JCIDS  operation  manual  provides  limited 
guidance  on  how  capability  sponsors  should  prioritize  the  gaps,  stating 
only  that  the  prioritization  should  be  based  on  the  potential  for  operational 
risk  associated  with  the  shortfalls.  The  JCIDS  manual  also  directs 
capability  sponsors  to  identify  redundancies  and  assess  whether  the 
overlap  is  operationally  acceptable  or  whether  it  should  be  evaluated  as 
part  of  the  trade-offs  to  satisfy  capability  gaps.  Three  of  the  12  validated 
ICDs  we  reviewed  from  fiscal  year  2010  did  not  address  redundancies. 
Furthermore,  only  one  of  these  ICDs  presented  to  the  JROC  in  fiscal  year 
2010  included  an  evaluation  of  the  overlaps.  The  JROC  did  not  address 
these  omissions  when  it  validated  the  documents. 


Conclusions 


In  the  last  several  years,  Congress  has  passed  legislation  to  give  the  JROC 
a  greater  role  in  prioritizing  military  requirements  and  shaping  sound 
acquisition  programs  by  encouraging  cost,  schedule,  and  performance 
trade-offs.  Taken  together,  these  steps  have  the  potential  to  improve  the 
affordability  and  execution  of  DOD’s  portfolio  of  major  defense 


^Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development 
System  (July  31,  2009). 

34While  the  JROC  validated  12  ICDs  during  fiscal  year  2010,  according  to  Joint  Staff 
officials,  an  additional  ICD  received  during  this  time  period  was  pending  final  certification 
as  of  January  2011. 

35Electronic  Health  Record  ICD  is  designed  to  address  10  gaps  in  the  Joint  Force  Health 
Protection  concept  of  operations,  including  comprehensive  medical  and  dental 
documentation,  inpatient  and  outpatient  order  entiy  and  management,  and  consult  and 
referral  management. 

command  and  Control  On-The-Move  ICD  is  the  capability  to  maintain  situational 
awareness  and  make  timely  and  informed  decisions  while  nonstationary.  It  is  designed  to 
address  six  gaps,  including  the  limited  capability  to  share  information  with  mission 
partners  and  an  inability  to  plan  collaboratively  while  on  the  move. 
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acquisition  programs.  However,  the  JROC  has  largely  left  prioritization 
and  trade-off  decisions  to  the  military  services,  despite  having  a  unique, 
joint  perspective,  which  would  allow  it  to  look  across  the  entire 
department  to  identify  efficiencies  and  potential  redundancies.  To  more 
effectively  leverage  its  unique  perspective,  the  JROC  would  have  to  change 
the  way  it  views  its  role,  more  regularly  engage  the  acquisition  community 
in  trade-off  discussions  at  early  acquisition  milestones,  and  more 
effectively  scrutinize  the  quality  of  the  resource  estimates  presented  by  the 
military  services.  Until  it  does  so,  the  JROC  will  only  be  a  marginal  player 
in  DOD’s  efforts  to  align  the  department’s  available  resources  with  its 
warfighting  requirements. 


Recommendations 
Executive  Action 


£q  To  enhance  the  JROC’s  role  in  DOD-wide  efforts  to  deliver  better  value  to 
the  taxpayer  and  warfighter,  we  recommend  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  chairman  of  the  JROC,  take  the  following  five 
actions: 


•  Establish  a  mechanism  to  review  the  final  AOA  report  prior  to 
Milestone  A  to  ensure  that  trade-offs  have  been  considered  and  to 
provide  military  advice  on  these  trade-offs  and  the  proposed  materiel 
solution  to  the  Milestone  Decision  Authority. 

•  Require  that  capability  sponsors  present  resource  estimates  that  have 
been  reviewed  by  a  military  service’s  cost  analysis  organization  to 
ensure  best  practices  are  being  followed. 

•  Require  that  capability  sponsors  present  key  results  from  sensitivity 
and  uncertainty  analyses,  including  the  confidence  levels  associated 
with  resource  estimates,  based  on  the  program’s  current  level  of 
knowledge. 

•  Assign  priority  levels  to  the  CDDs  based  on  joint  force  capability  gaps 
and  redundancies  against  current  and  anticipated  threats,  and  provide 
these  prioritization  levels  to  the  Under  Secretary  of  Defense 
(Comptroller)  and  the  military  services  to  be  used  for  resource 
allocation  purposes. 

•  Modify  the  JCIDS  operations  manual  to  require  that  CDDs  discuss 
potential  redundancies  across  proposed  and  existing  programs,  and 
address  these  redundancies  when  validating  requirements. 
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Agency  Comments 
and  Our  Evaluation 


The  Joint  Staff  provided  us  written  comments  on  a  draft  of  this  report.  The 
comments  are  reprinted  in  appendix  V.  The  Joint  Staff  also  provided 
technical  comments,  which  we  addressed  in  the  report,  as  appropriate. 

In  its  comments,  the  Joint  Staff  partially  concurred  with  all  five  of  our 
recommendations,  generally  agreeing  that  there  is  a  need  to  take  action  to 
address  the  issues  we  raised,  but  differing  in  terms  of  the  specific  actions 
that  should  be  taken. 

The  Joint  Staff  partially  concurred  with  our  recommendation  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff,  as  chairman  of  the  JROC,  establish  a 
mechanism  to  review  the  final  AOA  report  prior  to  Milestone  A  to  ensure 
that  trade-offs  have  been  considered  and  to  provide  military  advice  on 
these  trade-offs  and  the  proposed  materiel  solution  to  the  Milestone 
Decision  Authority.  The  Joint  Staff  noted  that  its  ongoing  review  of  JCIDS 
will  include  a  recommendation  that  AOA  results  be  briefed  to  FCBs. 
However,  the  FCB  will  only  elevate  these  briefings  to  the  JCB  or  JROC  on 
an  exception  basis.  The  Joint  Staff  explained  that  this  approach  would 
allow  the  JROC  to  provide  more  informed  advice  to  a  Milestone  Decision 
Authority  without  adding  another  round  of  staffing,  an  additional  JCIDS 
document,  or  an  official  validation  of  AOA  results.  We  agree  that  the  Joint 
Staff  should  seek  to  implement  this  recommendation  in  the  most  efficient 
and  effective  way  possible;  however,  given  our  finding  that  the  most 
significant  trade-off  decisions  are  made  as  a  result  of  an  AOA,  we  continue 
to  believe  that  the  results  should  be  reviewed  by  the  JROC. 

The  Joint  Staff  partially  concurred  with  our  recommendation  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  require  that  capability  sponsors 
present  resource  estimates  that  have  been  reviewed  by  a  military  service’s 
cost  analysis  organization  to  ensure  best  practices  are  being  followed.  The 
Joint  Staff  stated  that  program  office  cost  estimates  are  compared  to 
independent  cost  estimates  during  CDD  reviews.  However,  none  of  the 
seven  CDD  cost  estimates  we  reviewed  effectively  met  our  criteria  for  an 
independent  cost  estimate.  As  a  result,  we  believe  that  the  Joint  Staff 
needs  to  take  additional  action  to  ensure  that  resource  estimates 
presented  by  capability  sponsors  have  been  reviewed  by  a  military 
service’s  cost  analysis  organization.  The  Joint  Staff  also  stated  that  its 
ongoing  review  of  JCIDS  will  examine  how  to  highlight  this  area  during 
CDD  reviews. 

The  Joint  Staff  partially  concurred  with  our  recommendation  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  require  that  capability  sponsors 
present  key  results  from  sensitivity  and  uncertainty  analyses,  including  the 
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confidence  levels  associated  with  resource  estimates,  based  on  the 
program’s  current  level  of  knowledge.  The  Joint  Staff  stated  that  our 
recommendation  needs  further  study  to  understand  the  expected 
outcomes  and  the  required  authorities  for  the  JROC,  and  its  ongoing 
review  of  JCIDS  will  examine  how  to  highlight  this  area.  We  believe  that 
the  JROC  cannot  fully  consider  trade-offs  or  the  affordability  of  a 
proposed  program  unless  it  receives  information  on  the  risk  and 
uncertainty  associated  with  resource  estimates;  it  does  not  need  additional 
authority  to  require  capability  sponsors  to  present  the  results  of  this  type 
of  analysis  before  it  approves  proposed  requirements.  The  Joint  Staff  also 
noted  that  the  Director,  CAPE,  has  cost  analysis  responsibilities  for 
resource  estimates.  CAPE  cost  assessment  officials  reported  that  they 
rarely  participated  in  JCIDS  reviews.  As  a  result,  the  JROC  may  have  to  be 
more  proactive  in  reaching  out  to  CAPE  to  help  it  understand  the  risk  and 
uncertainty  associated  with  the  resource  estimates  it  receives. 

The  Joint  Staff  partially  concurred  with  our  recommendation  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  assign  priority  levels  to  CDDs  based 
on  joint  force  capability  gaps  and  redundancies  against  current  and 
anticipated  threats,  and  provide  these  prioritization  levels  to  the  Under 
Secretary  of  Defense  (Comptroller)  and  the  military  services  to  be  used  for 
resource  allocation  purposes.  The  Joint  Staff  agreed  that  the  identification 
of  joint  priorities  could  enhance  a  number  of  processes,  including  program 
and  budget  reviews.  It  noted  that  its  ongoing  review  of  JCIDS  will 
recommend  a  prioritization  framework  through  which  CDDs  will  inherit 
priority  levels  based  on  the  requirements  and  capability  gaps  identified  in 
ICDs  or  Joint  Urgent  Operational  Needs.  However,  the  Joint  Staff  argued 
against  prioritizing  based  on  CDDs  directly  because  it  would  provide  less 
flexibility.  We  believe  that  the  proposed  approach  could  be  effective  if  the 
Joint  Staff  addresses  the  inconsistencies  we  found  in  the  way  ICDs 
prioritize  gaps.  In  addition,  we  continue  to  believe  that  the  prioritization 
framework  should  facilitate  an  examination  of  priorities  across  CDDs. 

The  Joint  Staff  partially  concurred  with  our  recommendation  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  modify  the  JCIDS  operations  manual 
to  require  that  CDDs  discuss  potential  redundancies  across  proposed  and 
existing  programs,  and  address  these  redundancies  when  validating 
requirements.  The  Joint  Staff  stated  that  its  ongoing  review  of  JCIDS  will 
address  this  issue  by  establishing  unique  requirements  as  a  higher  priority 
than  unnecessarily  redundant  requirements,  and  by  establishing  a  post- 
AOA  review,  which  could  also  be  used  to  identify  unnecessary 
redundancies.  The  Joint  Staff  did  not  address  whether  it  would  update  the 
JCIDS  operations  manual  as  recommended  and  stated  that  reviewing 
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assessments  of  redundancies  in  CDDs  would  be  late  in  the  JCIDS  process. 
We  believe  that  potential  redundancies  should  be  discussed  at  multiple 
points,  including  during  CDD  reviews,  because  we  found  that  several  years 
can  pass  between  the  conclusion  of  an  AOA  and  this  review.  During  that 
time,  new  redundancy  issues  could  emerge. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Chairman  and  Vice  Chairman  of  the  Joint  Chiefs  of  Staff;  the  Secretaries  of 
the  Army,  Navy,  and  Air  Force;  and  the  Director  of  the  Office  of 
Management  and  Budget.  In  addition,  the  report  will  be  made  available  at 
no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-4841.  Contact  points  for  our  offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  Staff  members  making  key  contributions  to  this  report  are 
listed  in  Appendix  VI. 


Michael  J.  Sullivan 
Director,  Acquisition 

and  Sourcing  Management 
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Appendix  I:  Scope  and  Methodology 


To  conduct  our  work,  we  reviewed  relevant  sections  of  Title  10  of  the  U.S. 
Code,  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  (WSARA),  and 
the  National  Defense  Authorization  Act  for  Fiscal  Year  2008  to  establish 
the  role  of  the  Joint  Requirements  Oversight  Council  (JROC)  in 
considering  trade-offs  among  cost,  schedule,  and  performance  objectives; 
reviewing  the  estimated  level  of  resources  needed  to  fulfill  these 
requirements;  and  prioritizing  requirements.  We  also  reviewed  Department 
of  Defense  (DOD),  Joint  Staff,  and  military  service  guidance  documents,  as 
well  as  those  for  the  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  for  developing  and  validating  military  requirements,  to 
determine  how  these  roles  have  been  implemented  in  policy.  To  determine 
how  these  policies  have  been  implemented  in  practice,  we  analyzed 
information  and  capability  documents  contained  in  the  Joint  Staffs 
Knowledge  Management/Decision  Support  tool.1  To  do  so,  we  first 
established  how  many  requirements  documents — Initial  Capabilities 
Documents  (ICD),  Capability  Development  Documents  (CDD),  and 
Capability  Production  Documents  (CPD) — were  reviewed  by  the  JROC 
and  Joint  Capabilities  Board  (JCB)  during  fiscal  year  2010.  We  selected 
fiscal  year  2010  as  our  time  frame  because  WSARA  was  enacted  in  May 
2009,  and  this  would  allow  for  any  changes  the  JROC  would  implement  as  ^ 
result  of  this  legislation.  We  then  focused  our  analysis  on  the  unclassified 
requirement  documents  reviewed  by  the  JROC  and  JCB  in  fiscal  year  2010 
which  identified  capability  gaps  or  defined  performance  requirements  for 
new  major  defense  acquisition  programs:  13  ICDs  and  7  CDDs.  We 
assessed  these  documents,  as  well  as  briefings  presented  to  the  JROC  or 
the  JCB,  associated  meeting  minutes,  and  JROC  decision  memos.  We  also 
examined  15  JROC  reviews  of  programs  that  incurred  substantial  cost 
growth  after  program  start  in  fiscal  year  2010  to  determine  if  cost, 
schedule,  and  performance  trade-offs  were  made.  We  chose  this  time 
period  to  allow  for  any  changes  the  JROC  would  implement  as  result  of 
the  enactment  of  WSARA  in  May  2009. 

To  determine  the  extent  to  which  the  JROC  has  considered  trade-offs 
among  cost,  schedule,  and  performance  objectives  within  programs,  we 
reviewed  the  seven  CDDs  submitted  to  the  JROC  and  analyzed  the 


'This  system  is  designed  to  track  the  status  of  requirements  documents — ICDs,  CDDs, 
CPDs — submitted  to  the  JCIDS  process  for  review.  It  is  the  key  system  that  the  Joint  Staff 
and  other  DOD  components  use  to  review  and  comment  on  proposals.  To  assess  the 
reliability  of  the  data  system,  we  interviewed  Joint  Staff  officials  knowledgeable  about  the 
system,  and  cross-checked  information  in  the  system.  We  determined  that  the  data  were 
sufficiently  reliable  for  the  purposes  of  this  report. 
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Appendix  I:  Scope  and  Methodology 


information  presented  on  trade-offs.  We  focused  on  CDDs  because  they 
are  the  first  requirements  documents  that  contain  cost,  schedule,  and 
performance  objectives.  We  also  examined  JROC  decision  memos  to 
identify  whether  the  JROC  provided  input  on  cost,  schedule,  and 
performance  objectives  for  the  seven  proposed  programs  and  analyses  of 
alternatives  (AO A)  conducted  by  the  military  services  prior  to  JROC 
reviews.  We  also  met  with  officials  from  the  Joint  Staff;  Department  of  the 
Air  Force;  Department  of  the  Army;  Department  of  the  Navy;  Office  of  the 
Director  of  Cost  Assessment  and  Program  Evaluation  (CAPE);  Office  of 
the  Under  Secretary  of  Defense  (Comptroller);  Office  of  the  Assistant 
Secretary  of  Defense  for  Research  and  Engineering;  and  respective 
program  offices  about  these  issues.  To  obtain  combatant  command  views 
on  their  participation  in  the  joint  requirements  process  since  the 
implementation  of  WSARA,  we  developed  a  survey  administered  to  DOD’s 
10  combatant  commands.* * 3  The  survey  addressed  a  range  of  topics  related 
to  the  joint  requirements  process,  including  the  means  for  combatant 
commands  to  provide  information  on  their  capability  needs.  To 
understand  the  Joint  Staffs  ongoing  internal  JCIDS  review,  we  assessed 
the  review  charter  and  met  with  the  Joint  Staff  officials  managing  the 
review  to  discuss  the  recommendations  from  the  review  and  how  they 
might  affect  the  JROC’s  consideration  of  trade-offs.  We  also  observed  joint 
requirements  meetings  and  reviewed  prior  GAO  reports. 

To  determine  the  quality  and  effectiveness  of  efforts  to  estimate  the  level 
of  resources  needed  to  fulfill  joint  military  requirements,  we  assessed  the 
resource  estimates  used  to  support  the  seven  unclassified  proposed  major 
defense  acquisition  programs  reviewed  by  the  JROC  in  fiscal  year  2010 
against  the  best  practices  in  our  cost  estimating  guide.3  We  used  these 
criteria  to  determine  the  extent  to  which  these  resource  estimates  were 
credible,  well  documented,  accurate,  and  comprehensive.  We  scored  each 
best  practice  as  either  being  Not  Met — DOD  provided  no  evidence  that 
satisfies  any  of  the  criterion,  Minimally  Met — DOD  provided  evidence  that 
satisfies  a  small  portion  of  the  criterion,  Partially  Met — DOD  provided 
evidence  that  satisfies  about  half  of  the  criterion,  Substantially  Met — DOD 
provided  evidence  that  satisfies  a  large  portion  of  the  criterion,  and  Met — 


3Combatant  commands  are  DOD’s  operational  commanders.  There  are  six  combatant 

commands  with  geographic  responsibilities  and  four  with  functional  responsibilities. 
However,  in  August  2010,  the  Secretary  of  Defense  recommended  the  closure  of  the  U.S. 
Joint  Forces  Command. 

3GAO-09-3SP. 
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DOD  provided  complete  evidence  that  satisfies  the  entire  criterion.  We 
determined  the  overall  assessment  rating  by  assigning  each  individual 
rating  a  number:  Not  Met  =  1,  Minimally  Met  =  2,  Partially  Met  =  3, 
Substantially  Met  =  4,  and  Met  =  5.  Then,  we  took  the  average  of  the 
individual  assessment  ratings  to  determine  the  overall  rating  for  each  of 
the  four  characteristics.  To  perform  this  analysis,  we  obtained  and 
analyzed  program  resource  estimate  supporting  documentation,  including 
service  cost  positions,  technical  descriptions,  work  breakdown  structures, 
technology  readiness  assessments,  program  schedules,  and  AOA  reports. 
We  also  interviewed  program  and  cost  estimating  officials,  when 
necessary,  to  gather  additional  information  on  these  resource  estimates 
and  the  cost  models  used  to  produce  them.  Each  program  was  also 
provided  with  a  copy  of  our  assessment  of  their  resource  estimates  for 
review  and  comment. 

To  determine  the  extent  to  which  the  JROC  prioritized  requirements  and 
capability  gaps,  we  reviewed  the  13  ICDs  and  7  CDDs  submitted  to  the 
JROC  and  any  discussions  of  priorities  and  redundancies  contained  in 
each  document.  We  also  met  with  officials  from  the  Joint  Staff; 

Department  of  the  Air  Force;  Department  of  the  Army;  Department  of  the 
Navy;  and  Office  of  the  Under  Secretary  of  Defense  (Comptroller)  to 
discuss  the  extent  to  which  the  JROC  and  its  supporting  bodies  have 
addressed  prioritization  issues.  To  understand  the  Joint  Staffs  ongoing 
internal  JCIDS  review,  we  assessed  the  review  charter  and  met  with  the 
Joint  Staff  officials  managing  the  review  to  discuss  the  recommendations 
from  the  review  and  how  they  might  affect  the  JROC’s  prioritization  of 
requirements.  We  also  observed  joint  requirements  meetings  and  reviewed 
prior  GAO  reports. 

We  conducted  this  performance  audit  from  June  2010  to  June  2011  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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In  fiscal  year  2010,  the  Joint  Capabilities  Boards  (JCB)  and  Joint 
Requirements  Oversight  Council  (JROC)  combined  to  review  a  total  of  45 
new  requirements  documents,  including  11  that  were  classified,  2  that 
were  information  technology  programs,  and  8  documents  that  were  not 
associated  with  major  defense  acquisition  programs.  The  remaining  24 
requirements  documents  are  identified  in  figure  4.1 


'joint  Staff  officials  reported  that  in  fiscal  year  2010,  two  key  requirements  documents 
were  reviewed  by  a  Functional  Capabilities  Board,  but  not  a  JCB  or  JROC. 
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Figure  4:  Requirements  Documents  Reviewed  by  the  JCB  or  JROC  in  Fiscal  Year  2010 


Sponsor 

Capability  Gap 

Air  Force 

Advanced  Pilot  Trainer 

Air  Force 

Joint  Future  Theater  Lift 

Army 

Ground  Combat  Vehicle 

Army 

Joint  Direct-Support  Airborne  Intelligence,  Surveillance, 
and  Reconnaissance 

Army 

Positioning,  Navigation,  and  Timing  Assurance 

Defense  Intelligence  Agency 

Biometric  Enabled  Intelligence 

Defense  Intelligence  Agency 

Joint  Intelligence  Operations  Center 

Information  Technology  Enterprise 

Deputy  Assistant  Secretary  of  Defense  (Force  Health 
Protection  and  Readiness)  and  Joint  Staff  Surgeon/ 

J4  Health  Service  Support  Division 

Joint  Force  Health  Protection 

TRICARE  Management  Agency 

Electronic  Health  Record 

US  Joint  Forces  Command 

Command  and  Control  On-The-Move 

US  Joint  Forces  Command 

Global  Force  Management 

US  Northern  Command 

Maritime  Domain  Awareness  Joint  Integrating  Concept 

US  Strategic  Command  and  US  Joint  Forces  Command 

Joint  Aerial  Layer  Network 

Sponsor 

Proposed  Program 

Air  Force 

Common  Vertical  Lift  Support  Platform 

Air  Force 

HH-60  Recapitalization 

Army 

Army  Integrated  Air  and  Missile  Defense  System  of  Systems,  Increment  2 

Army 

Ground  Soldier  System,  Increment  1 

Navy 

Joint  Precision  Approach  and  Landing  System,  Increment  2 

Navy 

P-8A  Poseidon  Multi-mission  Maritime  Aircraft,  Increment  2  and  3 

Navy 

Ship  to  Shore  Connector 

Sponsor 

Existing  Program 

Air  Force 

Joint  Air-to-Surface  Standoff  Missile  -  Extended  Range  System,  Increment  1 

Army 

Apache  Block  III 

Marine  Corps 

Global  Combat  Support  System  -  Marine  Corps  /  Logistics  Chain 

Management 

Marine  Corps 

V-22  Block  C/20 

Source:  DOD's  Knowledge  Management  and  Decision  Support  Tool. 
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The  Joint  Requirements  Oversight  Council  (JROC)  conducted  15  reviews 
following  cost  breaches  in  fiscal  year  2010 — 6  Nunn-McCurdy  reviews  and 
9  tripwire  reviews.  Table  2  identifies  these  reviews. 


Table  2:  JROC  Cost  Breach  Reviews  in  Fiscal  Year  2010 


Nunn-McCurdy  reviews 

Apache  Block  III 

Advanced  Threat  Infrared  Countermeasure  /  Common  Missile  Warning  System 

DDG  1000 

Joint  Strike  Fighter 

Remote  Minehunting  System 

Wideband  Global  Satellite  Communication 

Tripwire  reviews 

Apache  Block  III 

Apache  Longbow  Block  II 

C-130  Avionics  Modernization  Program 

CH-47F 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals,  Increment  1 
Global  Hawk 

Patriot  Advance  Capability  3 
Tactical  Tomahawk 

Vertical  Take-off  and  Landing  Tactical  Unmanned  Air  Vehicle 

Source:  GAO  analysis  of  DOD  information. 
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Table  3  below  presents  the  best  practice  criteria  against  which  we 
assessed  the  resource  estimates  presented  to  the  Joint  Requirements 
Oversight  Council  during  fiscal  year  2010  Capability  Development 
Document  (CDD)  reviews. 


Table  3:  Best  Practice  Criteria  for  Resource  Estimates 


Characteristic 

Best  practices 

Comprehensive 

The  cost  estimate  includes  all  life  cycle  costs. 

The  cost  estimate  completely  defines  the  program,  reflects  the  current  schedule,  and  is  technically 
reasonable. 


The  cost  estimate  work  breakdown  structure  is  product-oriented,  traceable  to  the  statement  of  work/objective, 
and  at  an  appropriate  level  of  detail  to  ensure  that  cost  elements  are  neither  omitted  nor  double-counted. 

The  estimate  documents  all  cost-influencing  ground  rules  and  assumptions. 

Well-documented  The  documentation  should  capture  the  source  data  used,  the  reliability  of  the  data,  and  how  the  data  were 
normalized. 

The  documentation  describes  in  sufficient  detail  the  calculations  performed  and  the  estimating  methodology 
used  to  derive  each  element’s  cost. 

The  documentation  describes  step  by  step  how  the  estimate  was  developed  so  that  a  cost  analyst  unfamiliar 
with  the  program  could  understand  what  was  done  and  replicate  it. 

The  documentation  discusses  the  technical  baseline  description  and  the  data  in  the  baseline  is  consistent 
with  the  estimate. 

The  documentation  provides  evidence  that  the  cost  estimate  was  reviewed  and  accepted  by  management. 

Accurate  The  cost  estimate  results  are  unbiased,  not  overly  conservative  or  optimistic  and  based  on  an  assessment  of 

most  likely  costs. 

The  estimate  has  been  adjusted  properly  for  inflation. 

The  estimate  contains  few,  if  any,  minor  mistakes. 

The  estimate  is  based  on  a  historical  record  of  cost  estimating  and  actual  experiences  from  other  comparable 
programs. 

Credible  The  cost  estimate  includes  a  sensitivity  analysis  that  identifies  a  range  of  possible  costs  based  on  varying 

major  assumptions,  parameters,  and  data  inputs. 

A  risk  and  uncertainty  analysis  was  conducted  that  quantified  the  imperfectly  understood  risks  and  identified 
the  effects  of  changing  key  cost  driver  assumptions  and  factors. 

Major  cost  elements  were  crossed-checked  to  see  whether  results  were  similar. 

An  independent  cost  estimate  was  conducted  by  a  group  outside  the  acquiring  organization  to  determine 
whether  other  estimating  methods  produce  similar  results. 


Source:  GAO-09-3SP. 
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THE  JOINT  STAFF 

WASHINGTON.  DC 

7  June  2011 


Reply  ZIP  Code: 

20318-0300 

Mr.  Michael  J.  Sullivan 
Director,  Acquisition  and  Sourcing 
Management 

U.S.  Government  Accountability 
Office 

441  G  Street,  N.W. 

Washington,  D.C.  20548 

Dear  Mr.  Sullivan, 

Thank  you  for  the  opportunity  to  review  GAO  Draft  Report  GAO- 11 -502, 
“DOD  Requirements  Reviews:  Missed  Trade-off  Opportunities,”  dated  4  May 
2011  (GAO  Code  120916).  Please  find  enclosed  the  DOD  response  to  the 
report’s  recommendations. 

The  Joint  Staff  point  of  contact  is  Mr.  Randolph  Wood;  J-8;  703-614- 
9628,  randolph.wood@js.pentagon.mil. 


Sincerely, 


CRAIG  A.  FRANKLIN 
Major  General,  USAF 
Vice  Director,  Joint  Staff 
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ENCLOSURE 


GAO  DRAFT  REPORT  DATED  4  MAY  2011 
GAO-1 1-502  (GAO  CODE  120916) 

“DOD  REQUIREMENTS  REVIEWS:  MISSED  TRADE-OFF  OPPORTUNITIES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
ON  THE  GAO  RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommends  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  Chairman  of  the  Joint  Requirements  Oversight  Council 
(JROC),  establish  a  mechanism  to  review  the  final  analysis  of  alternatives  (AoA) 
report  prior  to  Milestone  A  to  ensure  that  trades-offs  have  been  considered  and 
to  provide  military  advice  on  these  trades-offs  and  the  proposed  material 
solution  to  the  milestone  decision  authority.  (See  page  27/ GAO  Draft  Report.) 

DOD  RESPONSE:  Partially  concur.  One  of  the  Joint  Capability  Development 
Process  Review  (JCDPR)  recommendations  is  to  have  AoA  results  and 
recommended  path  forward  briefed  to  the  appropriate  Functional  Capabilities 
Board  (FCB).  At  the  discretion  of  the  FCB  Chair,  by  exception,  the  briefing 
could  be  elevated  to  the  Joint  Capabilities  Board  (JCB)/JROC  for  further 
review.  This  recommendation  does  not  add  another  round  of  staffing,  an 
additional  Joint  Capabilities  Integration  and  Development  System  document, 
or  an  official  validation  of  the  AoA  results.  It  does  allow  the  JROC  to  provide 
more  informed  advice  to  the  milestone  decision  authority.  Concurrently, 
continued  participation  by  Joint  Staff  representatives  in  the  AoA  Senior 
Advisory  Group,  Executive  Steering  Committee,  and  Integrated  Product  Team 
will  ensure  that  timely  military  advice  on  trade-offs  will  be  considered  during 
the  execution  of  the  AoA. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  Chairman  of  the  JROC,  require  that  capability 
sponsors  present  resource  estimates  that  have  been  reviewed  by  a  military 
Service’s  cost  analysis  organization  to  ensure  best  practices  are  being  followed. 
(See  page  27/GAO  Draft  Report.) 

DOD  RESPONSE:  Partially  concur.  When  capability  development  documents 
(CDDs)  are  entered  into  staffing  for  Joint  Staff  review  and  approval,  a 
comparison  of  an  independent  cost  estimate  to  the  program  office’s  cost 
estimate  is  included.  A  method  to  highlight  and  examine  this  area  will  be 
incorporated  in  the  staffing  and  briefing  process  leading  to  a  JROC  validation, 
as  part  of  the  ongoing  JCDPR. 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  Chairman  of  the  JROC,  require  that  capability 
sponsors  present  key  results  from  sensitivity  and  uncertainty  analyses, 
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including  the  confidence  levels  associated  with  resource  estimates,  based  on 
the  program’s  current  level  of  knowledge.  (See  page  27/GAO  Draft  Report. 

DOD  RESPONSE:  Partially  concur.  The  Director,  Cost  Assessment  and 
Program  Evaluation  has  oversight  responsibility  for  analyses  of  alternatives 
and  cost  analyst  responsibilities  for  resource  estimates.  This  recommendation 
needs  further  study  to  understand  the  expected  outcomes  and  the  required 
authorities  for  the  JROC.  The  JROC  approves  joint  military  requirements  to 
inform  the  decision  points  of  the  acquisition  process.  However,  a  method  to 
highlight  and  examine  this  area  will  be  incorporated  in  the  staffing  and  briefing 
process  leading  to  a  JROC  validation  as  part  of  the  ongoing  JCDPR. 

RECOMMENDATION  4:  The  GAO  recommends  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  Chairman  of  the  JROC,  assign  priority  levels  to  the 
CDDs  based  on  joint  force  capability  gaps  and  redundancies  against  current 
and  anticipated  threats,  and  provide  these  prioritization  levels  to  the  Under 
Secretary  of  Defense  (Comptroller/Chief  Financial  Officer)  and  the  military 
Services  to  be  used  for  resource  allocation  purposes.  (See  page  27  through 
28/GAO  Draft  Report.) 

DOD  RESPONSE:  Partially  concur  on  the  prioritization  means.  One  of  the 
JCDPR  recommendations  is  a  prioritization  framework.  Details  are  still  being 
finalized,  but  the  current  recommendation  includes  prioritizing  capability 
requirements  to  provide  traceability  from  the  capability  requirement  to 
capability  gap  to  the  solution  being  pursued  (CDD).  Therefore,  the  CDDs  will 
“inherit”  priority  from  the  requirements  and  gaps  being  satisfied,  and  these 
requirements  and  gaps  may  come  from  one  or  more  Initial  Capability 
Documents  or  Joint  Urgent  Operational  Needs.  Priorities  based  upon  CDDs 
directly  would  be  a  less  flexible  construct  for  prioritization. 

DOD  RESPONSE:  Partially  concur  on  the  prioritization  usage.  The 
prioritization  framework  developed  under  the  JCDPR  has  been  designed  to 
provide  more  informed  decision  making  to  a  wide  variety  of  “consumers”  across 
the  Department.  Processes  that  could  be  enhanced  or  streamlined  by  the 
availability  of  joint  prioritization  include  the  Program  and  Budget  Review, 
combatant  command  Integrated  Priority  Lists  and  Capability  Gap  Assessments, 
and  Service  Program  Objective  Memorandum  generation.  In  addition, 
prioritization  could  inform  Joint  Experimentation  and  Science  and  Technology 
efforts. 

RECOMMENDATION  5:  The  GAO  recommends  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  Chairman  of  the  JROC,  modify  the  Joint  Capabilities 
Integration  and  Development  System  operations  manual  to  require  that  a  CDD 
discuss  potential  redundancies  when  validating  requirements.  (See  page  28/ 
GAO  Draft  Report.) 
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DOD  RESPONSE:  Partially  concur.  Reviewing  assessment  of  redundancies  in 
the  CDD  is  late  in  the  process  for  this  activity;  however,  a  method  to  examine 
this  area  during  CDD  staffing  and  briefing  will  be  addressed  as  part  of  the 
ongoing  JCDPR.  Two  of  the  recommendations  from  the  JCDPR  effort  address 
the  issue  of  unnecessary  redundancy  prior  to  the  generation  of  a  CDD.  One  of 
the  Joint  assessment  criteria  in  the  prioritization  framework  looks  specifically 
at  the  notion  of  requirement  uniqueness  or  redundancy.  Unique  requirements 
without  suitable  alternatives  gain  a  higher  priority  than  requirements  that 
would  be  unnecessarily  redundant  with  other  existing  requirements  or 
capability  solutions.  This  helps  encourage  the  drive  to  commonality  and 
collaboration  where  multiple  organizations  have  similar  or  identical 
requirements.  The  recommendation  for  a  post-AoA  review  provides  a  second 
opportunity  for  the  FCB  (or  JCB/ JROC)  to  review  the  alternatives  studied  and 
the  recommended  path  forward.  If  the  recommended  approach  introduces 
unnecessary  redundancy,  the  JROC  will  provide  appropriate  recommendations 
to  the  MDA  for  adjustments  to  the  path  forward. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 
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Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 
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